Added sigmoid like activation functions #9011

AIexanderDicke · 2023-08-25T09:23:58Z

Describe your change:

Add an algorithm?
Fix a bug or typo in an existing algorithm?
Documentation change?

Checklist:

rohan472000

For ensuring clarity and ease of parameter interpretation across the functions.

rohan472000 · 2023-09-06T07:41:41Z

neural_network/activation_functions/sigmoid_like.py

+    >>> np.linalg.norm(np.array([0.5, 0.66666667, 0.83333333]) - result) < 10**(-5)
+    True
+    """
+    return _base_activation(vector, 0, 1)


Suggested change

return _base_activation(vector, 0, 1)

return _base_activation(vector, alpha=0, beta=1)

rohan472000 · 2023-09-06T07:42:02Z

neural_network/activation_functions/sigmoid_like.py

+    >>> np.linalg.norm(np.array([0, 0.66666667, 1.6]) - result) < 10**(-5)
+    True
+    """
+    return _base_activation(vector, 1, beta)


Suggested change

return _base_activation(vector, 1, beta)

return _base_activation(vector, alpha=1, beta=beta)

rohan472000 · 2023-09-06T07:42:18Z

neural_network/activation_functions/sigmoid_like.py

+    >>> np.linalg.norm(np.array([0, 0.7310585, 0.462098]) - result) < 10**(-5)
+    True
+    """
+    return swish(vector, 1)


Suggested change

return swish(vector, 1)

return swish(vector, beta=1)

algorithms-keeper

Click here to look at the relevant links ⬇️

🔗 Relevant Links

Repository:

Contributing guidelines

Project Euler solution guidelines

Python:

Formatted string literals (f-strings)

Type hints

doctest

unittest

pytest

Automated review generated by algorithms-keeper. If there's any problem regarding this review, please open an issue about it.

algorithms-keeper commands and options

algorithms-keeper actions can be triggered by commenting on this PR:

@algorithms-keeper review to trigger the checks for only added pull request files

@algorithms-keeper review-all to trigger the checks for all the pull request files, including the modified files. As we cannot post review comments on lines not part of the diff, this command will post all the messages in one comment.

NOTE: Commands are in beta and so this feature is restricted only to a member or owner of the organization.

algorithms-keeper · 2023-09-07T15:01:27Z

neural_network/activation_functions/sigmoid_like.py

+import numpy as np
+
+
+def _base_activation(vector: np.ndarray, alpha: float, beta: float) -> np.ndarray:


As there is no test file in this pull request nor any test function or class in the file neural_network/activation_functions/sigmoid_like.py, please provide doctest for the function _base_activation

The examples that you have kept under comments, make it doctest, remove that Example word, and merge both statements i,e.. result and np.linalg....

>>> np.linalg.norm(np.array([0.5, 0.66666667, 0.83333333]) - ( _base_activation(np.array([0, np.log(2), np.log(5)]), 0, 1))) < 10**(-5) True

rohan472000 · 2023-09-07T16:28:01Z

neural_network/activation_functions/sigmoid_like.py

+import numpy as np
+
+
+def _base_activation(vector: np.ndarray, alpha: float, beta: float) -> np.ndarray:


The examples that you have kept under comments, make it doctest, remove that Example word, and merge both statements i,e.. result and np.linalg....

>>> np.linalg.norm(np.array([0.5, 0.66666667, 0.83333333]) - ( _base_activation(np.array([0, np.log(2), np.log(5)]), 0, 1))) < 10**(-5) True

tianyizheng02

Personally, I think it's better to implement the three activation functions without using a shared helper function. While it might not be as elegant, I think it's better from an educational standpoint for users to be able to see the explicit formula for each of the functions.

Also, we already have the sigmoid and SiLU in the maths/ directory. However, I'd rather we have these functions in neural_network/activation_functions like you did, so we should delete these two existing files in favor of yours.

tianyizheng02

Just small improvements, but other than that LGTM

tianyizheng02 · 2023-09-16T18:47:56Z

neural_network/activation_functions/sigmoid_like.py

+    >>> np.linalg.norm(np.array([0.5, 0.66666667, 0.83333333]) \
+        - sigmoid(vector=np.array([0, np.log(2), np.log(5)]))) < 10**(-5)


Suggested change

>>> np.linalg.norm(np.array([0.5, 0.66666667, 0.83333333]) \

- sigmoid(vector=np.array([0, np.log(2), np.log(5)]))) < 10**(-5)

>>> np.linalg.norm(np.array([0.5, 0.66666667, 0.83333333])

... - sigmoid(vector=np.array([0, np.log(2), np.log(5)]))) < 10**(-5)

I believe you can use ... to avoid using \

tianyizheng02 · 2023-09-16T18:48:11Z

neural_network/activation_functions/sigmoid_like.py

+        - sigmoid(vector=np.array([0, np.log(2), np.log(5)]))) < 10**(-5)
+    True
+    """
+    return 1 / (1 + np.exp(-1 * vector))


Suggested change

return 1 / (1 + np.exp(-1 * vector))

return 1 / (1 + np.exp(-vector))

Just slightly more concise

tianyizheng02 · 2023-09-16T18:48:51Z

neural_network/activation_functions/sigmoid_like.py

+    >>> np.linalg.norm(np.array([0.5, 1., 1.5]) \
+            - swish(np.array([1, 2, 3]), 0)) < 10**(-5)
+    True
+    >>> np.linalg.norm(np.array([0, 0.66666667, 1.6]) \
+            - swish(np.array([0, 1, 2]), np.log(2))) < 10**(-5)


Suggested change

>>> np.linalg.norm(np.array([0.5, 1., 1.5]) \

- swish(np.array([1, 2, 3]), 0)) < 10**(-5)

True

>>> np.linalg.norm(np.array([0, 0.66666667, 1.6]) \

- swish(np.array([0, 1, 2]), np.log(2))) < 10**(-5)

>>> np.linalg.norm(np.array([0.5, 1., 1.5])

... - swish(np.array([1, 2, 3]), 0)) < 10**(-5)

True

>>> np.linalg.norm(np.array([0, 0.66666667, 1.6])

... - swish(np.array([0, 1, 2]), np.log(2))) < 10**(-5)

tianyizheng02 · 2023-09-16T18:49:24Z

neural_network/activation_functions/sigmoid_like.py

+    >>> np.linalg.norm(np.array([0, 0.7310585, 0.462098]) \
+            - sigmoid_linear_unit(np.array([0, 1, np.log(2)]))) < 10**(-5)


Suggested change

>>> np.linalg.norm(np.array([0, 0.7310585, 0.462098]) \

- sigmoid_linear_unit(np.array([0, 1, np.log(2)]))) < 10**(-5)

>>> np.linalg.norm(np.array([0, 0.7310585, 0.462098])

... - sigmoid_linear_unit(np.array([0, 1, np.log(2)]))) < 10**(-5)

tianyizheng02 · 2023-09-16T18:49:39Z

neural_network/activation_functions/sigmoid_like.py

+            - sigmoid_linear_unit(np.array([0, 1, np.log(2)]))) < 10**(-5)
+    True
+    """
+    return vector / (1 + np.exp(-1 * vector))


Suggested change

return vector / (1 + np.exp(-1 * vector))

return vector / (1 + np.exp(-vector))

rohan472000 · 2023-09-23T10:00:05Z

#9078 is merged, now this build will pass, run it again.

adds sigmoid like activation functions

0575cfa

rohan472000 suggested changes Sep 6, 2023

View reviewed changes

feedback

63943a7

algorithms-keeper bot added require tests Tests [doctest/unittest/pytest] are required awaiting reviews This PR is ready to be reviewed labels Sep 7, 2023

algorithms-keeper bot reviewed Sep 7, 2023

View reviewed changes

adds tests for _base_activation

6cb23f9

algorithms-keeper bot removed the require tests Tests [doctest/unittest/pytest] are required label Sep 7, 2023

AIexanderDicke requested a review from rohan472000 September 7, 2023 15:09

rohan472000 suggested changes Sep 7, 2023

View reviewed changes

tianyizheng02 reviewed Sep 7, 2023

View reviewed changes

removes _base_activation

8f37073

AIexanderDicke requested review from tianyizheng02 and rohan472000 September 9, 2023 07:40

rohan472000 approved these changes Sep 9, 2023

View reviewed changes

tianyizheng02 requested changes Sep 16, 2023

View reviewed changes

feedback

8de62d9

algorithms-keeper bot added the tests are failing Do not merge until tests pass label Sep 17, 2023

AIexanderDicke requested a review from tianyizheng02 September 17, 2023 09:11

tianyizheng02 approved these changes Sep 27, 2023

View reviewed changes

cclauss closed this Sep 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added sigmoid like activation functions #9011

Added sigmoid like activation functions #9011

AIexanderDicke commented Aug 25, 2023

rohan472000 left a comment

rohan472000 Sep 6, 2023

rohan472000 Sep 6, 2023

rohan472000 Sep 6, 2023

algorithms-keeper bot left a comment

algorithms-keeper bot Sep 7, 2023

rohan472000 Sep 7, 2023

rohan472000 Sep 7, 2023

tianyizheng02 left a comment

tianyizheng02 left a comment

tianyizheng02 Sep 16, 2023

tianyizheng02 Sep 16, 2023

tianyizheng02 Sep 16, 2023

tianyizheng02 Sep 16, 2023

tianyizheng02 Sep 16, 2023

rohan472000 commented Sep 23, 2023

	return _base_activation(vector, 0, 1)
	return _base_activation(vector, alpha=0, beta=1)

	return _base_activation(vector, 1, beta)
	return _base_activation(vector, alpha=1, beta=beta)

		import numpy as np


		def _base_activation(vector: np.ndarray, alpha: float, beta: float) -> np.ndarray:

		>>> np.linalg.norm(np.array([0.5, 0.66666667, 0.83333333]) \
		- sigmoid(vector=np.array([0, np.log(2), np.log(5)]))) < 10**(-5)

	return 1 / (1 + np.exp(-1 * vector))
	return 1 / (1 + np.exp(-vector))

		>>> np.linalg.norm(np.array([0, 0.7310585, 0.462098]) \
		- sigmoid_linear_unit(np.array([0, 1, np.log(2)]))) < 10**(-5)

	return vector / (1 + np.exp(-1 * vector))
	return vector / (1 + np.exp(-vector))

Added sigmoid like activation functions #9011

Added sigmoid like activation functions #9011

Conversation

AIexanderDicke commented Aug 25, 2023

Describe your change:

Checklist:

rohan472000 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

algorithms-keeper bot left a comment

Choose a reason for hiding this comment

🔗 Relevant Links

Repository:

Python:

Automated review generated by algorithms-keeper. If there's any problem regarding this review, please open an issue about it.

algorithms-keeper actions can be triggered by commenting on this PR:

algorithms-keeper bot Sep 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tianyizheng02 left a comment

Choose a reason for hiding this comment

tianyizheng02 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rohan472000 commented Sep 23, 2023